Use ROCm clang for Ghost unified-memory examples by dannybaths · Pull Request #190 · amd/HPCTrainingExamples

dannybaths · 2026-09-08T08:01:33Z

What

The five MPI_Ghost_Exchange_Ver[1-5] test scripts now select the ROCm C and C++
compilers from $ROCM_PATH/bin when amdclang and amdclang++ are present there.
Each script exports CC/CXX immediately before its per-test CMake configure, so
the GhostExchange examples build with ROCm clang instead of whatever the MPI wrapper
defaults to (often GCC on Cray/Open MPI login environments).

These examples use #pragma omp requires unified_shared_memory. GCC does not
implement that clause on the offload path these tests need.

Why

On MI300A systems where the MPI module defaults to GCC, CMake picks GCC and every
Ghost Ver1–5 build aborts on the requires unified_shared_memory line before launch.
The scripts already use ROCM_PATH for the ROCm runtime; selecting the matching
ROCm compiler keeps the tests self-contained without launcher-specific -x flags or
Open MPI-only options on the scripts' srun branch.

Validation

MI300A / gfx942, Open MPI, HSA_XNACK=1, ROCM_PATH pointing at a ROCm install
with bin/amdclang and bin/amdclang++. Focused run on upstream main (20e93223):

git clone https://github.com/amd/HPCTrainingExamples.git
cd HPCTrainingExamples/tests
export HSA_XNACK=1
export ROCM_PATH=/path/to/rocm
rm -rf build && cmake . -B build
ctest -V -R '^MPI_Ghost_Exchange_Ver[1-5]$' --test-dir build

Environment details from the validation jobs:

AAC7 focused proof, jobs 10626 unpatched and 10627 patched:

PrgEnv-amd-openmpi/openmpi-5.0.10-ofi-7.14.0
rocm/7.14.0
craype-accel-amd-gfx942
cray-python/3.12.12
ROCM_PATH=/shareddata/opt/rocm-7.14.0
HSA_XNACK=1
PRTE_MCA_rmaps_default_mapping_policy=:OVERSUBSCRIBE
mpirun=/shareddata/opt/rocmplus-ompi-7.14.0/openmpi-5.0.10-libfabric2.3.1-xpmem-2.7.4/bin/mpirun

In the unpatched AAC7 tree, CC, CXX, and FC were unset before CTest. The
per-test CMake runs selected /usr/bin/cc and /usr/bin/c++, identified as
GNU 11.5.0. With this PR applied, the per-test CMake runs selected
/shareddata/opt/rocm-7.14.0/bin/amdclang and
/shareddata/opt/rocm-7.14.0/bin/amdclang++, identified as Clang 23.0.0.

AAC6 no-regression, job 19736:

module system: Lmod 8.6.19
loaded modules: rocm/7.14.0, openmpi/5.0.10-ucc1.6.0-ucx1.19.1-xpmem-2.7.4, openblas/0.3.33, magma/2.10.0, pytorch/2.9.1
ROCM_PATH=/nfsapps/ubuntu-24.04/opt/rocm-7.14.0
amdclang/amdclang++=/nfsapps/ubuntu-24.04/opt/rocm-7.14.0/bin/amdclang{,++}
CDASH_SUBMIT=0

Static checks on the branch:

git diff --check
bash -n tests/mpi_ghost_exchange_ver{1,2,3,4,5}.sh

Unpatched upstream: 0/5 pass. All five builds use GCC and fail at compile with
sorry, unimplemented: 'unified_shared_memory' clause on 'requires' directive not supported yet.

With this change: 5/5 pass. Per-test CMake logs show
Check for working C/CXX compiler: …/rocm-…/bin/amdclang{,++}; CTest reports
100% tests passed, 0 tests failed out of 5.

AAC7 proof: current-main control on ROCm 7.14 / Open MPI 5.0.10 — unpatched 0/5,
patched 5/5.

AAC6 no-regression: MI300A, Lmod rocm/7.14.0, Slurm job 19736 on
ppac-pl1-s24-16 — 5/5 PASS with amdclang/amdclang++ under $ROCM_PATH/bin.

gcapodagAMD · 2026-09-14T14:59:23Z

I am not seeing the issue arising on aac7 @dannybaths

gcapodagAMD · 2026-09-14T15:04:59Z

it looks like I am not picking up gcc but CrayClang

gcapodagAMD · 2026-09-14T15:05:37Z

what modules are you loading when testing this?

dannybaths · 2026-09-14T16:05:38Z

you are right @gcapodagAMD, I indeed forgot to list the environment. Sorry for that.
I have modified the main test above by adding this:

PrgEnv-amd-openmpi/openmpi-5.0.10-ofi-7.14.0
rocm/7.14.0
craype-accel-amd-gfx942
cray-python/3.12.12
ROCM_PATH=/shareddata/opt/rocm-7.14.0
HSA_XNACK=1
PRTE_MCA_rmaps_default_mapping_policy=:OVERSUBSCRIBE
mpirun=/shareddata/opt/rocmplus-ompi-7.14.0/openmpi-5.0.10-libfabric2.3.1-xpmem-2.7.4/bin/mpirun_

Signed-off-by: Daniele Bagni <danieleb@xilinx.com>

gcapodagAMD · 2026-09-14T18:44:31Z

   fi
   if [ -z "$FC" ]; then
      export FC=`which ftn`
   fi


I think changes should be made here because this covers the "CRAY" path, in the "AMD" case we have the right compilers set with the amdclang module

gcapodagAMD · 2026-09-14T18:46:07Z

can you tell me what is the scope of the module PrgEnv-amd-openmpi ? Should this load any craype modules? also, being PrgEnv-amd should it already point you to using amdclang and amdclang++ instead of gcc? if so I think the change should be in the module itself and maybe as a safeguard also in that place in the script that I mentioned above 👆

Ghost Ver1-Ver5 previously exported the ROCm amdclang++ whenever it existed, which also overrode the Cray wrapper choice made by the CRAYPE_VERSION / /etc/cray-release branch above. On a Cray PE login that is a regression: measured on an MI300A node with ROCm 7.14, the stock scripts pass all five requested ghost CTests through the CC wrapper (CrayClang), while the unconditional override loses Ver3, Ver4 and Ver5. The failure the previous commit addressed happens in the other state: /etc/cray-release exists on a Cray system even when no PrgEnv module puts the wrappers on PATH, so the branch above exports an empty CXX and CMake falls back to /usr/bin/c++, which does not implement the OpenMP requires unified_shared_memory directive. In that state all five tests fail unpatched and pass with the ROCm compilers. Fill in the ROCm compilers only when CXX is empty or does not resolve, so a working wrapper choice or an explicit user setting is left alone.

dannybaths · 2026-09-14T18:58:56Z

Thanks for pushing back on this one — your CrayClang observation turned out to be the whole story, and the patch as I first wrote it was too broad.

I re-proved it on AAC7 against current main (a257bf6e, so on top of your ghost cleanup) in two module states, MI300A node, ROCm 7.14.0, Slurm job 11053:

State	unpatched `CMAKE_CXX_COMPILER`	unpatched	patched, old unconditional form
A — stock `PrgEnv-cray/8.7.0`	`/opt/cray/pe/craype/2.7.36/bin/CC`	Ver1–Ver5 PASS	Ver1, Ver2 PASS; Ver3, Ver4, Ver5 FAIL
B — `PrgEnv-amd-openmpi/openmpi-5.0.10-ofi-7.14.0`	`/usr/bin/c++`	Ver1–Ver5 FAIL	Ver1–Ver5 PASS

So on a Cray PE login the stock scripts already make the right choice and my override was costing three ghost tests, which is exactly what you were seeing. The failure I originally reported is real, but it is confined to state B: /etc/cray-release exists on this machine even when no PrgEnv module has put the compiler wrappers on PATH, so the CRAYPE_VERSION || /etc/cray-release branch runs, which CC yields nothing, CXX is exported empty, and CMake falls back to /usr/bin/c++ — which does not implement #pragma omp requires unified_shared_memory.

I have force-pushed a narrowed version (rebased onto a257bf6e). It no longer overrides anything; it only fills the gap the Cray branch leaves behind:

if [ -z "${CXX:-}" ] || ! command -v "${CXX}" >/dev/null 2>&1; then
   if [ -x "${ROCM_PATH:-}/bin/amdclang++" ] && [ -x "${ROCM_PATH:-}/bin/amdclang" ]; then
      export CXX="${ROCM_PATH}/bin/amdclang++"
      export CC="${ROCM_PATH}/bin/amdclang"
   fi
fi

In state A command -v resolves the wrapper, so the block is a no-op and CrayClang is kept. In state B CXX is empty, so the ROCm compilers are used. It is also a no-op when ROCM_PATH is unset, and it leaves an explicit user CXX alone. I moved it up next to the existing compiler-selection block, since that is what it completes, instead of leaving it just before cmake.

So far I have verified the narrowed form with a shell trace on the edited scripts in both states. The full two-state CTest re-proof is queued on AAC7 and on AAC6, pinned to the same ROCm 7.14.0 so the two machines compare directly, and I will post those numbers here when they land.

Module environment for both states

State A, stock Cray PE:

craype-x86-genoa
libfabric/2.3.1
craype-network-ofi
perftools-base/26.03.0
cce/21.0.0
craype/2.7.36
cray-mpich/9.1.0
cray-libsci/26.03.0
PrgEnv-cray/8.7.0
rocm/7.14.0
craype-accel-amd-gfx942
cray-python/3.12.12
ROCM_PATH=/shareddata/opt/rocm-7.14.0
HSA_XNACK=1
which CC  -> /opt/cray/pe/craype/2.7.36/bin/CC
mpicc     -> /opt/cray/pe/mpich/9.1.0/ofi/cray/20.0/bin/mpicc (craycc ... -lmpi_cray)
rocminfo  -> XNACK enabled: YES

State B, OpenMPI PrgEnv:

craype-x86-genoa
libfabric/2.3.1
craype-network-ofi
perftools-base/26.03.0
rocm-new/7.14.0
craype-accel-amd-gfx942
rocm/7.14.0
openmpi/5.0.10-libfabric2.3.1-xpmem-2.7.4-rocm-7.14.0
cray-python/3.12.12
PrgEnv-amd-openmpi/openmpi-5.0.10-ofi-7.14.0
ROCM_PATH=/shareddata/opt/rocm-7.14.0
HSA_XNACK=1
which CC  -> not present in PATH
which cc  -> /usr/bin/cc
mpicc     -> .../openmpi-5.0.10-libfabric2.3.1-xpmem-2.7.4/bin/mpicc (gcc ... -lmpi)
rocminfo  -> XNACK enabled: YES

Compiler in both states: AMD clang version 23.0.0git (ROCm/llvm-project 46fcb339 +PATCHED:440716f8).

gcapodagAMD · 2026-09-14T20:37:38Z

does state B show anything if you do echo $CRAYPE_VERSION ?

gcapodagAMD · 2026-09-14T20:38:37Z

+# the compiler wrappers on PATH.  The branch above then exports an empty CXX and
+# CMake falls back to /usr/bin/c++, which does not implement
+# "#pragma omp requires unified_shared_memory".  Use the ROCm compilers only in
+# that case, so a working wrapper or an explicit user choice is left alone.


please remove these comments

gcapodagAMD · 2026-09-14T20:39:34Z

+# the compiler wrappers on PATH.  The branch above then exports an empty CXX and
+# CMake falls back to /usr/bin/c++, which does not implement
+# "#pragma omp requires unified_shared_memory".  Use the ROCm compilers only in
+# that case, so a working wrapper or an explicit user choice is left alone.


please remove these all over

gcapodagAMD · 2026-09-14T20:41:04Z

@dannybaths careful I think you are still using the mpi wrappers with gcc, you also need to set OMPI_CXX etc to amdclang++ etc

gcapodagAMD · 2026-09-14T20:42:14Z

load the full B path then do mpicc -show and see if it says amdclang if not you are missing those env variables

gcapodagAMD · 2026-09-14T22:10:59Z

@dannybaths let me know if after the above the problem still shows, if so I'll merge. thanks

Use ROCm clang for Ghost unified-memory examples

d5b13b0

Signed-off-by: Daniele Bagni <danieleb@xilinx.com>

gcapodagAMD reviewed Sep 14, 2026

View reviewed changes

dannybaths force-pushed the fix-ghost-usm-rocm-compiler branch from 4d58d1e to 61a35e7 Compare September 14, 2026 18:56

gcapodagAMD reviewed Sep 14, 2026

View reviewed changes

Navigation Menu

Uh oh!

Use ROCm clang for Ghost unified-memory examples - #190

Use ROCm clang for Ghost unified-memory examples#190
dannybaths wants to merge 2 commits into
amd:mainfrom
dannybaths:fix-ghost-usm-rocm-compiler

dannybaths commented Sep 8, 2026 •

edited

Loading

Uh oh!

gcapodagAMD commented Sep 14, 2026

Uh oh!

gcapodagAMD commented Sep 14, 2026

Uh oh!

gcapodagAMD commented Sep 14, 2026

Uh oh!

dannybaths commented Sep 14, 2026

Uh oh!

gcapodagAMD Sep 14, 2026

Uh oh!

gcapodagAMD commented Sep 14, 2026 •

edited

Loading

Uh oh!

dannybaths commented Sep 14, 2026

Uh oh!

gcapodagAMD commented Sep 14, 2026

Uh oh!

gcapodagAMD Sep 14, 2026

Uh oh!

gcapodagAMD Sep 14, 2026

Uh oh!

gcapodagAMD commented Sep 14, 2026

Uh oh!

gcapodagAMD commented Sep 14, 2026

Uh oh!

gcapodagAMD commented Sep 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

dannybaths commented Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Validation

Uh oh!

gcapodagAMD commented Sep 14, 2026

Uh oh!

gcapodagAMD commented Sep 14, 2026

Uh oh!

gcapodagAMD commented Sep 14, 2026

Uh oh!

dannybaths commented Sep 14, 2026

Uh oh!

gcapodagAMD Sep 14, 2026

Choose a reason for hiding this comment

Uh oh!

gcapodagAMD commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dannybaths commented Sep 14, 2026

Uh oh!

gcapodagAMD commented Sep 14, 2026

Uh oh!

gcapodagAMD Sep 14, 2026

Choose a reason for hiding this comment

Uh oh!

gcapodagAMD Sep 14, 2026

Choose a reason for hiding this comment

Uh oh!

gcapodagAMD commented Sep 14, 2026

Uh oh!

gcapodagAMD commented Sep 14, 2026

Uh oh!

gcapodagAMD commented Sep 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dannybaths commented Sep 8, 2026 •

edited

Loading

gcapodagAMD commented Sep 14, 2026 •

edited

Loading